Data Access Reorganizations in Compiling Out-of-Core Data Parallel Programs on Distributed Memory Machines

This paper describes optimization techniques for translating out-of-core programs written in a data parallel language like HPF to message passing node programs with explicit parallel I/O. We rst discuss how an out-of-core program can be translated by extending the method used for translating in-core programs. We demonstrate that straightforward extension of in-core compilation techniques does not work well for outof-core programs. We then describe how the compiler can optimize the code by (1) determining appropriate le layouts for out-of-core arrays, (2) permuting the loops in the nest(s) to allow e cient le reading/writing, and (3) partitioning the available incore node memory among references based on I/O cost estimation. Both our analytical approach and experimental results indicate that these optimizations can reduce the amount of time spent in I/O by as much as an order of magnitude. Performance results on IBM SP-2 are presented and discussed.